Apparatus and Method for Recognition of Suspicious Activities

ABSTRACT

A medication confirmation method and apparatus. The method of an embodiment of the invention includes the steps of capturing one or more video sequences of a user administering medication by a video capture device, storing the captured one or more video sequences to a non-transitory memory, and analyzing at least one of the stored video sequences to determine one or more indications of suspicious activity on behalf of the user.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is Continuation In Part application of U.S. patentapplication Ser. No. 13/235,387, filed Sep. 18, 2011 to Hanina et al.,titled APPARATUS AND METHOD FOR RECOGNITION OF PATIENT ACTIVITIES,currently pending, which claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/498,576, filed Jun. 19, 2011 to Hanina et al.,titled APPARATUS AND METHOD FOR RECOGNITION OF PATIENT ACTIVITIES,currently pending, the entire contents of each of these applicationsbeing incorporated herein by reference. This application is also aContinuation in Part Application of U.S. patent application Ser. No.12/815,037, filed Jun. 14, 2010 to Hanina et al., titled APPARATUS ANDMETHOD FOR RECOGNITION OF PATIENT ACTIVITIES WHEN OBTAINING PROTOCOLADHERENCE DATA, currently pending, which claims the benefit of U.S.Provisional Patent Application Ser. No. 61/331,872, filed May 6, 2010,to Hanina et al., titled APPARATUS AND METHOD FOR RECOGNITION OF PATIENTACTIVITIES WHEN OBTAINING PROTOCOL ADHERENCE DATA, the entire contentsof each of these applications being incorporated herein by reference.This application is also a Continuation In Part application of U.S.Patent Application Ser. No. 13/189,518, filed Jul. 24, 2011 to Hanina etal., titled METHOD AND APPARATUS FOR MONITORING MEDICATION ADHERENCE,currently pending, which claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/495,415, filed Jun. 10, 2011 to Hanina et al.,titled METHOD AND APPARATUS FOR MONITORING MEDICATION ADHERENCE, theentire contents of each of these applications being incorporated hereinby reference.

FIELD

This invention relates generally to patient compliance in medicationadministration protocol scenarios including ingestion of medication, andmore particularly to an apparatus and method for the collection,analysis and transmission of data related to patient movements relatedto such medication administration and ingestion of the medication intothe body of a user, placing the medication in their mouth, or othermethod of transferring medication into their body, including featuresfor identifying suspicious or purposefully malicious activity on thepart of the patient.

BACKGROUND

Dr Lars Osterberg, M. D. and Dr, Terence Blaschke have reported in theNew England Journal of Medicine, Adherence to Medication, (N Engl J Med2005; 353:487-97) 2005 an alarming lack of adherence to requiredmedication protocol, further noting that while the average rates ofadherence in clinical trials is categorized as “high”, this number stillcomprises only rates of 43 to 78 percent. Most importantly, the authorsnote “The ability of physicians to recognize nonadherence is poor, andinterventions to improve adherence have had mixed results.” Adherence,p. 487. The authors conclude “Poor adherence to medication regimens iscommon, contributing to substantial worsening of disease, death andincreased healthcare costs.” Adherence, p. 494. The Trend Repot Series,2008 Patient Adherence Update: New Approaches for Success, October 2008,report similar discouraging statistics. This broad range may possiblycontribute to the public confidence in the FDA approval process and theimportance of continued surveillance of a drug throughout the process.Furthermore, it may help to explain why, according to the Journal of theAmerican Medical Association (JAMA May 1, 2002), one out of every fivenew drugs that comes to market in the US is found to have serious orlife-threatening adverse effects—unknown or undisclosed at the time ofapproval. It is against this backdrop of poor adherence, and potentialdanger to patients, that the present invention operates.

It has been widely recognized that methods and systems for insuringproper medication ingestion or administration by individuals are veryimportant in defending against unnecessary sickness, deaths and otherproblems. Giving instructions and then letting patients fend forthemselves has been shown not to work particularly well. This is becauseit is not only the improper ingestion of medicines that is the primarycause of medical danger. Rather, an overall lack of sufficient patientguidance is also part of the problem. Further, the inability to confirma proper prescription regimen being provided to a user in the firstplace may cause a number of other problems with the use of suchmedication. As has been shown in regards to various public healthmedication administration situation, such as administration oftuberculosis medication by the WHO, Directly Observed Treatment (DOT)improves compliance of patients. Global Tuberculosis Control: A ShortUpdate to the 2009 Report, World Health Organization, 2009. As is shownin this report, funding for implementing DOT programs is high. Thus, theability to implement such a program with less of a financial burdenwould be desirable.

Traditionally, participants attend introductions and follow ups forclinical trials in-person. Other patients outside of the clinical trialsetting attempting to adhere to a particular medication protocolsimilarly are given a prescription and a particular set of instructionsfrom a prescribing medical provider or prescribing doctor. Compliance isthen measured at a next visit with that prescribing professional throughtraditional methods of counting unused medication, and patientinterviews. Thus, data collection is similarly limited to patientvisits, rather than on a daily basis. These old methods such as patientquestioning and medication counting have been proven to be inadequatemeasures of adherence and offer no information on dose timing and drugholidays (omission of medication for three or more sequential days).

Compliance and medication adherence technologies can increase thestatistical power of clinical trials. Through the use of suchtechnology, clinical events can be precisely linked to medication usehistory. Captured data can be linked to other sources such as EDC,patient diaries and data collected by the physician. Technologies cancreate many possibilities for remote visits and data capture. Whilesmart packaging technologies exist such as RFID-enabled computer chiptechnology, smart blister packs and MEMS caps (microprocessor in abottle cap), they are: a) invasive and need to be physically attached tothe medications; b) are non-conclusive regarding compliance—a patientmay activate the technology without ingestion of the medication; c)remain largely unadopted in clinical trials by the pharmaceutical andbiotech companies due to their high cost; and d) take a longer time toimplement. Further, electronic patient diaries allow for ease of entryof data by a patient. These diaries, however, are still subject toissues related to compliance with medication adherence. Thus, even if apatient is meticulous about entering information into the diary, andthus complying with the requirements for data entry, there is still noguarantee that they are properly taking medication at prescribed times.This problem is even more acute when a participant is performing asuspicious action, or is otherwise maliciously avoiding taking theirmedication.

Jo Carol et al. stated that “The most reliable method for researchpurposes, although not practical in a clinical setting, may be acombination approach that includes pill counts, patient self-report, andelectronic monitoring.” (Carol J. et al, Patterns to AntiretroviralMedication, The Value of Electronic Monitoring, AIDS, 17 (12), pp1,763-767, October 2003. To date, technologies alone have only been usedto monitor compliance rather than to encourage it. These technologiesalso provide no specific defense against patients who are purposefullynot taking their medication.

A number of systems exist that provide instructions to a user regardingwhen to take a medication and records when the user indicates that amedication has been taken. U.S. Pat. No. 7,359,214 describes such asystem. A device is provided that provides instruction to a patientregarding medications to take. Furthermore, the system may provide amethod for determining that the prescription is appropriate given thepatient's conditions, and other medications he or she may already betaking. The system may monitor the dispensing of medicine in accordancewith a predetermined treatment protocol. While such a system providesmany improvements for easing a burden on the patient, this systemsuffers in many ways and in particular in ways relevant to theadministration of clinical trials and other active patient monitoring ofmedication adherence.

Most importantly, this system provides no mechanism for actuallyconfirming that a patient is in fact properly administering requiredmedication, including placing a medication pill into their mouth, orinjecting or inhaling medication following a predetermine series ofsteps as required in a clinical drug trial, as prescribed by aprescribing physician in the case where adherence to a particularregimen may prove to be critical to efficacy of the prescriptionregimen, in various public health scenarios, in situations where failureto keep up a prescription regimen can potentially harm a population as awhole, such as the generation of antibiotic-resistant bacteria strains,in various disease management scenarios, or in home care situationswhere maintaining proper control of administering healthcareprofessionals is critical. Further, while the system may be sufficientfor one who is in full possession of their mental faculties, anyindividual who may have difficulty following directions, or one who isactively avoiding medication may still not be taking required medicationafter it is dispensed. Thus, participants may be forgetful, visuallyimpaired, purposefully avoiding taking their medication, or otherwise donot believe in the benefit of taking such medication, and may thus notproperly log medication administration. Furthermore, the system requirespreloading of various medications into a dispenser, and thus likelyrequires regular visits by an administering manager to be sureappropriate medications are in fact properly loaded therein. It issurely possible that an inexperienced user may place incorrectmedications into the device, or may somehow provide incorrect dosagesinto the device. Additionally, for potentially more complex regimens,there is no method provided for insuring that a user is able to followsuch a protocol, and to thereafter confirm that the user has in facttaken all required medications in accordance with any providedinstructions or the like, or has taken the medications according to oneor more specifications or followed suggested procedures. Furthermore,this system is expensive and requires constant maintenance to confirmthat the various mechanical parts are in working order. Finally, thereis no consideration of participants who may be trying to trick thesystem and purposefully avoid taking their medication as may be the casein various patient populations such as schizophrenia populations, prisonpopulations, or other groups.

U.S. patent application Ser. No. 11/839,723, filed Aug. 16, 2007, titledMobile Wireless Medication Management System provides a medicationmanagement system employing mobile devices and an imaging technology sothat a user is able to show a pill to be taken to the system, and thesystem can then identify the medication. Patient histories are availableto an administrator, including various vital signs as measured by thesystem. Images may also be taken of the patient, provider, medicationcontainer or the like. While the system professes to ensure adherence toa protocol, the system only provides such help if requested by a user.There is in fact no particular manner in which to ensure actualadherence, including ingestion, inhalation, injection of the medication,or the relationship of adherence to the efficacy or safety of the drugover time. When requiring adherence to a predetermined protocol for aclinical trial, this is particularly relevant.

Additionally, existing systems fail to maintain an audit trail for postadministration review by a medical official or other clinical trialadministrator, and further cannot therefore confirm confirmation ofproper medication administration or population management. This onceagain thus fails to defend against suspicious or malicious medicationusers who purposefully do not take their medications.

Therefore, it would be desirable to provide an apparatus that overcomesthe drawbacks of the prior art.

SUMMARY

In U.S. patent application Ser. No. 12/620,686, filed Nov. 18, 2009,titled Method and Apparatus for Verification of MedicationAdministration Adherence; currently pending, U.S. patent applicationSer. No. 12/646,383, filed Dec. 23, 2009, titled Method and Apparatusfor Verification of Clinical Trial Adherence, currently pending; U.S.patent application Ser. No. 12/646,603, filed Dec. 23, 2009, titledMethod and Apparatus for Management of Clinical Trials, currentlypending; and U.S. patent application Ser. No. 12/728,721, filed Mar. 22,2010, titled Apparatus and Method for Collection of Protocol AdherenceData, currently pending, the contents of these four applications beingincorporated herein by reference, the inventors of the present inventionhave proposed a system, method and apparatus that allow for completecontrol and verification of adherence to a prescribed medicationprotocol or machine or apparatus use in a clinical trial setting,whether in a health care provider's care, or when self administered in ahomecare situation by a patient.

These applications present the only medication management system thatmay determine whether a user is actually following a protocol, provideadditional assistance to a user, starting with instructions preferablyincluding one or more interactive and real-time audio, visual textual orthe like prompts based upon one or more actions detected of the user,and moving up to contact from a medication administrator if it isdetermined that the user would need such assistance in any medicaladherence situation, including clinical trial settings, home caresettings, healthcare administration locations, such as nursing homes,clinics, hospitals and the like, and in clinical trial settings.

The inventive solution builds on these initial inventions and providesone or more features that may be employed in accordance with thesesystems to detect suspicious activity of a user, and to thwart users whopurposefully do not take their medication. The inventive system mayvisually and audibly recognize a fixed series of actions, eachcomprising part of the medication administration process, and mayfurther recognize deviations from the fixed series of actions, whetheraudio, video, and whether used for monitoring orally ingested,injectable, inhalable or other administered medications. The system mayalso interact in real-time with the patient when something is identifiedas incorrect, wrong or suspicious and either prompt a different actionor raise an alert and recommend an intervention by a healthcareprovider.

In accordance with an embodiment of the present invention, a motioncapture procedure for capturing motion information related to theadministration of pill or film based oral medications, or injectable,inhaler-based, other non-pill based medication, or any other form ofpatient administration task that may be performed, may be utilized inaccordance with one or more of the inventions noted in theabove-referenced applications. Therefore, in accordance with anembodiment of the present invention, a method and apparatus may beprovided for analyzing captured patient motion data, preferably in nearreal time to provide feedback to the user, to determine a number oftimes a participant performs some action that is considered suspicious.Additionally, the patient performance information may be analyzedasynchronously to determine other features of data that may suggest somemalicious intent on the part of the patient.

Further in accordance with an embodiment of the present invention, oneor more predetermined motion sequences may be determined, and one ormore related errors or other suspicious activities may be definedrelated to each of these one or more motion sequences. The automatedsystem of the invention may then monitor a number of times these one ormore suspicious activities may take place during a medicationadministration, and over multiple administrations in order to determinea pattern of potentially suspicious behavior over time.

Still other objects and advantages of the invention will in part beobvious and will in part be apparent from the specification anddrawings.

The invention accordingly comprises the several steps and the relationof one or more of such steps with respect to each of the others, and theapparatus embodying features of construction, combinations of elementsand arrangement of parts that are adapted to affect such steps, all asexemplified in the following detailed disclosure, and the scope of theinvention will be indicated in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the invention, reference is made tothe following description and accompanying drawings, in which:

FIG. 1 is a flow chart diagram depicting a method in accordance with anembodiment of the invention;

FIG. 2 is a flowchart diagram depicting a video sequence capture methodin accordance with an embodiment of the invention;

FIG. 3 is a depiction of a position determining process in accordancewith an embodiment of the invention;

FIG. 4 is a depiction of another position determining process inaccordance with an embodiment of the invention;

FIG. 5 is a depiction of a motion tracking process in accordance with anembodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The invention will now be described making reference to the followingdrawings in which like reference numbers denote like structure or steps.Referring to FIG. 1, a data flow overview in accordance with theoperation of an embodiment of the present invention is shown. Inaccordance with this embodiment of the invention, information about aparticular drug to be the subject of a clinical trial, to be employed ina public health or disease management situation, or the like, othermedication administration program or prescription, or other patientself-administered medical task such as performing a home based urinetest or the like may be provided in a database 105, and existingindustry medication information databases 110 are preferably employed toaccess prescription, interaction, application, and other availableinformation about any number of proposed prescription andnon-prescription medications and their possible interaction with theclinical trial or other medications. Further, patient medical records115 may be used, and as will be described below, in conjunction with theindustry medical information and a medical professional's prescribingexpertise to confirm that a patient is a good candidate for such aclinical trial, or medication administration program. These databasesmay be accessed in a manner known to one of ordinary skill in the art.This information may further include information related to associatedlikely typical elements of suspicious or malicious behavior. Suchelements may be based upon the type of medication, method of medicationadministration, demographics or other information about the patient,other features, or a combination of these elements. Suspicious activityinformation 118 is preferably transferred to the system at this time aswell, and as noted above may include one or more indications of patientmovements or other actions that may indicate suspicious behavior onbehalf of the patient, a malicious intent to trick the system, oractions of other high risk population. The various embodiments of thepresent invention may also be provided to determine if a patient isperforming an action or task correctly or if they are simply making amistake.

Once confirmed, a medication administration and ingestion or othermedication regimen in accordance with the clinical trial or otherprescription requirements such as in a public health, medical practiceenvironment or the like may be prescribed and entered into the system ofthe invention at 120. Such medication administration regimen may includeone or more of ingesting or swallowing a pill, using an inhaler device,using an injectable medication device, employing a patch, sublingualadministration, a cheek or other skin located medication administrationdevice or the like. Once entered into the system, a particularprescription regimen may cause a set of user instructions, varioustraining sequences and the like 125 to be generated and transmitted toan apparatus provided to a patient in accordance with an embodiment ofthe invention for access to the system of the invention. Such anapparatus may comprise a custom designed video and audio capture,analysis and transmission apparatus, a smart phone or other mobiledevice including a camera or other video and audio capture apparatuses,a netbook, laptop computer, desktop computer, tablet device or the like,free standing, or built into a mirror or cabinet or the like, or othercomputing appliance allowing for the display of instructions to apatient, and allowing for the eventual capture, analysis andtransmission of video, audio and other analysis information. Wheninstalling software on a user's own hardware system, it is preferredthat the software detect and otherwise test or determine that thehardware attempting to be utilized by the patient is sufficient toimplement the invention and is sufficient to run a software packageprovided in accordance with the invention. Thus, the software may checkthat a camera includes sufficient resolution, that a memory of thedevice is of sufficient size to allow for sufficient captured videostorage, that audio may be properly captured, and that the transmissionsystem includes sufficient bandwidth to transmit and receive capturedvideo, audio, video instructions and the like. Processing may also beperformed at a remote location, thus allowing the user to include alighter application or the like on their local device. Alternatively,the user may employ the local device as a gateway only, all data beingtransmitted to a remote location for processing, and returning responsesas a result of such processing. Thus, a user may be able to dial up avideo conference number, or otherwise interact with a remote site, suchas by visiting a particular website or URL.

Such user instructions and training sequences may include generalinstructions about the particular medication subject to the currenttrial or medication administration protocol, methods for administration,warnings about side effects, and concerns about drug interactions withcommon substances or medications, or other medications prescribed to thepatient by the system or by another medical service provider. It iscontemplated in accordance with an embodiment of the invention that suchset of user instructions may be interactive, allowing a user to viewadditional information about such instructions or prescriptions asdesired. These instructions may comprise written, audio or videoinstructions provided to the user on a display of the user apparatus. Itis further contemplated that such instructions may indicate one or moremovement sequences to be associated with a corresponding one or moremedication administration sequences. These instructions preferablyindicate proper and improper motions that may be taken by a patient foringesting a pill, using an inhaler, using an injectable medication, andthe like, and may indicate various motions that may or may not beperformed by a user. In such a manner, the patient may be instructed toproperly perform all requested actions, and avoid actions that may beindicative of a patient trying to trick the system. In an alternativeembodiment, it may be possible to particularly not describe to thepatient the various suspicious behaviors that will be tracked in anattempt to “trap” the unsuspecting malicious patient. These traps may bemaintained and running in a background of the system. Thus, as will bedescribed below, detection of one or more errors may generate real timedisplayed video and/or audio feedback to the patient in order to correctactions, while one or more other errors may be logged by the system butpreferably provide no feedback to the patient, and thus may beaccumulated and used to analyze patient actions without providinginstructions and help to avoid being caught by the system. After being“caught” a number of times, a state of the user may be changed to one ofheightened security awareness. Thus, a user may be first started in alow level of observation. After a predetermined number of potentiallyactions that are caught, the user may be labeled as a user that isattempting to trick the system, or that is performing one or moresuspicious acts, thus warranting a heightened level of security.Additional suspicion will surround review of all actions of the patient.After properly using the system correctly for another predeterminedperiod of time, the user may be returned to the initial normal state. Ifon the other hand, suspicious activity continues, the user may be movedto yet another state where the user may be recommended to be removedfrom a clinical trial, or taken off a particular medication, forexample.

In accordance with one or more embodiments of the present invention, oneor more of these sets of motions or actions may comprise confirmationthat a user has placed a pill in their mouth and has properly swallowedthe pill correctly and has therefore ingested the medication properlythrough visual confirmation of location, confirmation that the user hasproperly used an inhaler device through visual confirmation of positionand/or audio confirmation of actuation, confirmation that the user hasproperly used an injectable device through visual confirmation ofposition and/or audio confirmation of actuation, or the like.

Referring to the lower portion of FIG. 1, the horizontal line indicatesa time for patient administration of medication. At such time, the usermay be notified to take their medication through any desirablecommunication and notification system, including text messaging, email,telephone call, automated calendar reminder or the like. While notexplicitly shown, first, preferably the identity of a user may byconfirmed through the use of a facial recognition sequence, otherbiometric identification sequence, voice recognition, other passwordidentification system, or a combination thereof. Other features of useof the system, such as time to turn on, time to perform other tasks,etc. may be recorded to further aid in determining a consistent identityof a user of the system. The user may also be asked to say “hello” orsome other word sequence to allow for audio recognition of the voice ofthe user. Upon recognition of the individual, the system may display oneor more data regarding the individual, such as, by way of example only,name, patient status, medication to be administered, calendar indicatingto the patient when medication has been administered and if anyadministration times have been missed, and, selectively, a scoreindicative of a level of compliance of the individual with themedication protocol, if desired. Other metrics may also be tracked, suchas if too many skips have been registered by a user, or if too many selfreports that the user has taken their medication without the user of theautomated system, each indicating potentially suspicious activity. Thisidentification information may be stored to a remote location to aid indetermining whether a particular user is registering at more than onesite, with more than one device, and thus attempting to be paid twicefor the trial. Such patients are unlikely to be interested in taking themedication at all. Additional attributes of a patient may also be storedin a centralized database or the like to potentially alert otherclinical trials of potential offenders. Thus, one or more patientbehavior profiled, facial identification characteristics voicerecognition, combination thereof, may allow one or more subjects to beplaced or a watch list or the like after having been identified asnon-compliers. Thus, patients trying to get medication from multipleclinics, patients trying to sign up for multiple clinical trials, orpatients trying to sign up at multiple sites in a single clinical trialmay be recognized and determined before being able to be engaged atthese other sites.

The system shall preferably be able to learn the pattern of suspiciouspatients based on all data collected (such as number of misses, numberof skips, manual ratios, time on tasks, number of usability errors,number of suspicious errors, study coordinator/sponsors notes, flags,video reviewing results.). Then a patient can be easily classified assuspicious level 1, level 2 . . . or not by checking whether the patientfits in certain pattern. The machine will keep learning each day whenmore data are gathered and more patients are enrolled. Thus, a moreaccurate placement of the user in a particular state in a state machine,a will be described below, can be more accurately determined.

Once identified and notified of a type of medication to be administered,the patient may display a medication, such as a pill, dissolvable filmor the like, administration apparatus, such as an inhaler, injectableapparatus, or other medication form (including a pill bottle or thelike) to confirm that the medication is correct and is the currentlyprescribed medication to be taken through the use of text recognition,medication recognition, barcode or other code reading of one or moreunique identifiers from the administration apparatus, pill bottle or thelike, or other appropriate medication recognition scheme. One or moreconfidence level measurements may be employed, such as that described inco-pending application Ser. No. 13/110,500 filed May 18, 2011 to Guan etal., titled Method and Apparatus for Pattern Tracking, the entirecontents thereof being incorporated herein by reference.

In addition to recognizing the pill or other medication, the system maypreferably track how patient holds the pill or other medication, andfurther may track continuous motion from hand to mouth to confirm thatno suspicious action has taken place. Thus, the system may also checkconsistent hand usage between identification of the pill and placing thepill in the mouth of the user. In addition to tracking the motion of thehand, if the user is to take the pill out of a blister pack, the usermay track a motion sequence for tracking complete gestures on screenfrom taking out of the pill bottle or blister pack all the way toplacing in mouth/ingesting, drinking water, showing and tracking watergo down, swallowing, and even showing empty mouth so that any deviationfrom the desired sequence may be identified as a potentially suspiciousactivity. Furthermore, the system may view the blister pack to determineif the correct number of pills have been removed, that the correctnumber of pills are remaining, and whether other pills or the like havebeen removed that should not have been removed. Such suspicious activitymay be used to automatically flag potentially suspicious activity.

Thereafter, the patient may be instructed to administer the medicationin the prescribed manner. One or more of these administration sequencesmay comprise confirmation that a user has placed a pill in their mouth(ingested the pill) through visual confirmation of location,confirmation that the user has properly used an inhaler device throughvisual confirmation of position and/or audio confirmation of actuation,confirmation that the user has properly used an injectable devicethrough visual confirmation of position and/or audio confirmation ofactuation. Thus, for an oral administration, the user may be asked toplace the pill in their mouth, show the pill in their mouth to thecamera to confirm its presence, drink a glass of water, and show anempty mouth to the camera to confirm the pill is gone. The user may beasked to change the position of their tongue relative to the pill toprovide a more complete view of the pill in the mouth of the user to thecamera. For example, if the user only presents the top of their tongueto the camera, and refuses to show under their tongue, a suspiciousactivity may be flagged. Improper tilting of the glass of water, or thehead of the user when drinking the water may be further automaticallyrecognized and provided as potentially suspicious activity, thuschanging the state of the user and perhaps changing a level of scrutinyfor reviewing activity of the user. Swallowing motions may also beconfirmed, the absence thereof providing further evidence of potentiallysuspicious or malicious behavior. For an inhaler, the user may be askedto properly position the inhaler and show the positioning to the camera,to actuate the inhaler in view of the camera, to breathe in view of thecamera, and to hold their breath for a predetermined period of time infront of the camera. Different sounds, for example different frequenciesemitted from the inhaler, may suggest that the user may be blocking theaerosol or power of the inhaler with their teeth, either in error or onpurpose. Such incorrect positioning may suggest incorrect placement, andtherefore incorrect medication administration. For an injectablemedication the user may be asked to show the medication to the camera,to place the injectable medication adjacent the proper body part toreceive the injection in view of the camera, to actuate the injectablemedication in view of the camera, and the like.

During this administration, monitoring of the various indicatedpotentially suspicious activities may be provided. Thus, as describedabove, a determination of whether the head or face of the patient leavesthe field of view of the camera may be determined, and as will bedescribed below, a state of the user may be changed based upon this orother suspicious activities noted in accordance with this application.Other suspicious activities that may be tracked may include the patientcovering their mouth with their hand during medication administration ofa pill based medication, other body movements indicating an attempt toremove the medication or the like. Furthermore, audio recognition mayalso be employed to determine whether the patient is spitting out apill, for example, or to assist in determining whether an inhalable orinjectable medication has been properly actuated. Each of these featuresmay be preferably monitored over time so that while a single indicationof potentially suspicious behavior provides important information,continued performance of such suspicious behavior will provide a morecomplete picture of attempts to trick the system, thus allowing forintervention to identify the patient, and address the suspiciousbehavior. Different actions performed on the part of the user may allowthe user to be classified into one or more states, the state beingchanged based upon one or more actions they may perform over time. Sucha state machine may be performed in accordance with one or more featuresas described in copending U.S. patent application Ser. No. 13/189,516,filed Jul. 24, 2011 to Hanina et al., titled Method and Apparatus forMonitoring Medication Adherence, the contents thereof being incorporatedby reference. The medication administration states noted therein may beemployed to classify suspicious users based upon their actions.

If a medication is being used to treat a certain symptom, therefore inaccordance with one or more embodiments of the invention, the system maybe employed to measure visually if that symptom is improving. Forexample, concentration, speed to complete tasks, motor control etc. Ifsymptoms or behavior (eye movement still erratic for example) fail toimprove, this may suggest that the user is not taking their medication.Mood may also be measured and monitored as a signal of whether the useris taking their medication. Various actions may once again be used toclassify a user in one or more medication administration states, thussuggesting a level of review and follow up that may be required by thesystem to follow up with the user.

Additionally, determination of movement of the imaging device may bedetermined, through analysis of information provided in the field ofview thereof, or through the use of one or more gyroscopes oraccelerometers thereon. Thus, if the user tampers with the position ofthe imaging device or camera, it will be known and the monitor orhealthcare provider may be automatically notified. Other clues, such aschanges in a background setting at one or more critical times ofmedication administration, or other indications of movement of thedevice, or the like.

After confirmation or failure of confirmation of such administration,the patient may be provided with a progress report regarding how theyhave performed over time, and further providing encouragement for futureadherence. Additionally, notice of a next administration time may beprovided, along with one or more messages from a healthcare providerregarding protocol changes, or other desired information. Furthermore,various detected instances of potentially suspicious behavior may bereported to a healthcare provider or the like via a dashboard reportinggeneral adherence information along with the noted suspicious activityinformation. A warning may also be visually shown to the patient thatunusual activity has been detected. They may be told that incorrect useof the medication or protocol may result in their studycoordinator/healthcare provider being contacted, their being removedfrom the trial, or there may be an additional level of monitoringapplied to the patient.

Therefore, in accordance with the invention, confirmation of patientadherence to the prescribed administration schedule for the medicationas prescribed by the clinical trial or other prescription regimen may bedetermined, while suspicious or malicious behavior may be identified.While such confirmation may take a number of forms, in accordance withthe invention, a preferred method for such confirmation may includecapturing a video and audio sequence of the patient actuallyadministering the medication. In a further preferred method, such asequence for such confirmation may include employing a facialrecognition sequence or other biometric confirmation that a particularpatient is in fact receiving treatment, but may also provide for theability to obscure the face or other identifying feature of a user,simplify a displayed image through rotoscoping or the like, or otherwiseencrypt such information to allow for the storage and use of such imageswhile protecting the identity of the patient, a technique that may bebeneficial when a medication administration manager is providing ageneral report about a clinical trial, and not trying to remedy asituation with a particular patient, or in particular in a public healthor disease management scenario. Activity recognition, gesturerecognition, computer vision processing or other feature for determiningwhether a particular subject movement meets a predefined movementsequence may be employed to be sure that the patient is properly takingprescribed medication. This same gesture recognition may also beemployed to determine suspicious or malicious behavior on the part ofthe user, as described above. Audio recognition may also be employed todetermine suspicious noises such as coughing noises or the like at oneor more critical key moments associated with medication administration.For example if coughing consistently occurs routinely after the user haspace a medication pill in their mouth, then the patient may be escalatedto a state providing for a higher level of scrutiny of review during theadministration process over time. Finally, measurement of time on task,indicative of an amount of time required for the user to perform one ormore prescribed steps may also be employed to aid in determiningsuspicious behavior. This time on task may be measured across manydifferent devices and instances of use of the apparatus to determine oneor more trends that may be interesting, and may be used to determinesuspicious behavior. For example, time on task may be employed todetermine miming of an activity that the user I not really performingthe medication administration steps as required.

Referring next to FIG. 2, a method in accordance with an embodiment ofthe present invention for performing audio and video capture andrecognition of adherence to a prescribed protocol, and for determiningone or more indications of suspicious or malicious activity, isdescribed, as set forth in steps 130 and 135 of FIG. 1. In FIG. 2, apatient may first log into the system of the invention at step 205,employing the facial recognition, biometric recognition, password entry,voice recognition, or other patient identification method, and at step210 proper medication is preferably confirmed as noted above, throughthe use of bar code reading, text recognition, visual recognitionemploying video or still image recognition, or other medicationrecognition technique as described above. The patient may be reminded tolog onto the system to take their medication through any type ofreminder, such as a text message, email, phone call, automated alarm orthe like. Processing then passes to step 225 where the user may beprompted to perform one or more predetermined actions, the video and/oraudio of each of these actions preferably being captured. Video captureanalysis may then begin at step 230, such analysis comprising analysisof the newly captured video and/or audio. At step 235 it may bedetermined whether the action has been properly captured, and whetherthe captured action has been properly analyzed by the system in one ormore manners as will be described below.

If it is determined that administration of the medication did not takeplace properly, processing may return to step 225 and the user may beonce again prompted to perform the action. (Of course, if this processinvolves actual administration of pill, film, inhaler, injectablemedication, or any other medication, it may not be proper to requestre-performance of the action, unless it can be determined that the userdid not actually administer the medication.) If the action has beenproperly performed and is able to be analyzed, processing ends at step250 where the various captured video sequences are stored. These storedsequences may also be made available for human review and involvement,when it is determined that this would be beneficial, and may furtherpreferably be employed to determine suspicious or malicious activity.Areas of interest for review may be highlighted and marked aspotentially including suspicious or incorrect activity. Percentageestimates of risk may be automatically provided by the system based onrisk algorithms that are generated by the state machine, and based upona state into which the user may have been placed based upon their prioractivity, and one or more learned activities from a plurality of users.

Analysis of the captured video in order to determine suspicious ormalicious behavior will now be described in greater detail. As is shownin FIG. 3, a box 410 may be provided on a display viewable by a patientusing the system. Indicator box 410 may further be provided as acircular or other appropriate indicator. Indicator box 410 may befurther hidden from the view of the user. Alternatively a bezel or otheredge of a smartphone, tablet computer, other mobile device, computer orthe like may be used as a reference, i.e. as the “box.” A representationof the patient's face may be shown in a position relative to an optimalfilming position for the use of the system. Thus, while facialrepresentation 400 a is properly positioned, facial representation 400 bis positioned to the left of indicator box 410, and facialrepresentation 400 c is positioned down and to the right of indicatorbox 410. The inventors of the present invention have determined that oneindicator of suspicious or malicious behavior is the face of the patientleaving the field of view either a great number of times, and/or at oneor more crucial times. Thus, if the patient's face leaves the displayafter placing a pill in their mouth, for example, but before showingthat they have swallowed the pill, it may be that the patient isspitting out the pill. If their face leaves the display for one day,this may simply be user, but if it is determined that the patient's faceis leaving the display on a consistent basis over time, suspicious ormalicious behavior may be determined. Therefore, positions 400 b and 400c may be designated as suspicious positions, and that upon this positionbeing captured for a particular user at critical times over a number ofadministrations may result in flagging the participant for furtherreview. Thus, an automated system for determining possible suspicious ormalicious behavior is provided. The system may report such activity, oralternatively flag the patient to be more carefully looked after goingforward.

In an alternative embodiment, a healthcare provider may have a reason tosuspect a particular patient is acting suspiciously or maliciously,either by reviewing a recorded video of the patient taking theirmedication, or for any other reason. In this situation, it iscontemplated in accordance with this embodiment of the invention thatthe healthcare provider is preferably able to indicate that the patientshould be subject to an increased scrutiny. This indication may be madethrough a dashboard in which the healthcare provider can select a radiobutton or the like thus indicating this increased level of scrutiny, orotherwise make such an indication. Once flagged for additional scrutiny,a threshold for automatically determining suspicious or maliciousbehavior, such as the number of times a patient is able to have theirface leave the screed before a warning is provided to the healthcareprovider. Of course, this increased scrutiny may be applied to any ofthe suspicious or malicious activity recognition sequences described inaccordance with this application, or to any other recognition sequencesthat may be employed.

Other visual motions or gestures that may be monitored for suspiciousactivities may include the moving of a hand to the patient's mouth aftershowing a pill in the mouth of the patient, indicating that the patientmay be removing the pill. Once again, a one off situation may not beproblematic, while consistent movement of the hand of the user in frontof the mouth at a particularly sensitive time may warrant additionalscrutiny, and a warning about suspicious or malicious behavior. Othermovements of one or more body parts of the patient may also be included.Furthermore, audio indication of performing suspicious or maliciousactivity may be monitored, such as monitoring the user spitting out thepill. Such monitoring may automatically be performed based upon audioinformation, video information, or both. Other indications of maliciousactivity may include tracking a pill spit or thrown by the user across ascreen, etc. Additionally, the pill or other medication may appeardeformed to the camera. This may be from misuse or tampering by theuser., and may indicate melting, tampering, cutting or opening the pillor tablet, r even emptying a capsule of its contents of medicationbefore ingestion. Alignment of the portions of a capsule may also beanalyzed. Thus, any noted deformation, color change, marking change orthe like may be automatically determined by the system and reported toone or more monitors or key stakeholders.

If a monitor or other healthcare provider is provided a video review ofone or more actions performed by a user of the system, it iscontemplated in accordance with one or more embodiments of the inventionthat the monitor may be provided with a continuous one or more videoclips for review. It is contemplated in accordance with one or moreembodiments of the invention that if the system determines one or morevideo clips may include one or more suspicious activity actions, thesevideo clips may be annotated automatically by the system, a riskestimate preferably being provided by the system, and correlated inaccordance with a state of the user, the risk profile changing as theuser moves through one or more different user states. Upon review ofthese video clips by the monitor or other healthcare provider, the usermay further annotate the video by indicating further information along atimeline of video timestamps. This annotation is preferably furtherstored with the video for others to see, or for further analysis by thecomputer system in accordance with one or more embodiments of theinvention. Based upon these annotations, and one or more otherindicators of suspicious or malicious activity, one or more reports maybe generated and one or more key stakeholders may be notified. Thus, theuser may be classified into one or more categories based upon theiractivity, and based upon these categories, reported or further monitoredas noted above.

In addition to monitoring the movement of body parts, it is contemplatedin accordance with the invention to monitor, either alone or inconjunction with the movement of the body parts, movement of one or moremedication administration devices, such as an inhaler or injectormechanism. This may similarly apply to a cup or the like foradministering a liquid medication, or a container for holding one ormore pill, film or other oral medications. Therefore, as is shown inFIG. 4, in addition to tracking positioning of the patient, movement ofone or more objects indicative of suspicious or malicious behavior,either absolutely or relative to another body part, may be determined.In the particular embodiment shown in FIG. 4, the positioning of aninhaler, glass of water, etc. relative to the display area, mouth, orface of the user may be employed. Similar tracking may be employed foran injectable medication delivery device, in accordance with yet anotheralternative embodiment of the invention. As is shown in FIG. 4, aninhaler 500 may be determined to be properly positioned in a box 522,the box being green, for example, as in the description of FIG. 4. Ofcourse, the box need not be shown to the patient, in the manner notedabove with respect to FIG. 3. Such an object, however, is more likely tobe improperly positioned not only left to right and up to down, but alsoin distance to the imaging apparatus. Thus, in addition to a patientmoving the apparatus off the display side to side or up and down, thepatient may also move away from the imaging device in an attempt totrick the system. In such a manner the patient may be attempting tocapitalize on one or more limitations of the imaging device, such as theresolution thereof, low light positions, and the like, and any affectsuch resolution might have on the ability of the imaging device toidentify shape, color text or other coding, or the like associated withthe object being imaged. Thus, if positioned too far away from theimaging apparatus, a sequence of boxes 510, 511, 512 and a smallrepresentation of inhaler 500 may be provided to alert the user to movethe inhaler closer, or may be employed, while invisible to the patient,to determine whether an action has been properly determined assuspicious or malicious. Thus, if it is determined that the inhaler isnot only too far away, but off center, boxes 520, 521, 522 may beprovided (either visibly or not) to tack the position of the inhaler todetermine any potential suspicious or malicious movements on the part ofthe patient.

Similar functionality may be provided for monitoring the position of aninjectable apparatus relative to a user body part and display of theimaging apparatus to receive the injection, including relative angle anddistance to the body part. Thus, as with the facial movement noted abovewith respect to FIG. 3, movement of the medication administrationapparatus (inhaler or injectable) off the display (and thus out of viewof the imaging apparatus), or away from the imaging apparatus to a pointwhere the details of the medication administration apparatus are nolonger discernible, may be monitored, and if performed multiple timesover a predetermined timeframe, suspicious or malicious intention may bedetermined.

Furthermore, as is shown in FIG. 5, in addition to using absolutepositioning of a medication administration apparatus, it is contemplatedin accordance with the various embodiments of the invention that themovement of a medication administration apparatus 700 may be tracked,and thus any suspicious or malicious looking movements of the apparatus700 may be determined. Thus, if the patient moves the apparatus tooquickly, towards the edge of a field of view, towards an incorrect bodypart, faces the apparatus away from their mouth, or the like, suspiciousor malicious behavior may be determined even thought the apparatusremains in the field of view at all times. Thus, as is show in FIG. 5,an administration apparatus 700 is indicated to be reoriented from ahorizontal to a vertical orientation through movement in the directionnoted by arrows A. A set of guidance tracks 710 a, 710 b may bedisplayed to a patient, or may be maintained apart from the patientdisplay, and successive apparatus positions and orientations may bedetermined relative to the guidance tracks. As the apparatus moves alongthe proscribed path, proper location and orientation may be determined,while looking for any irregular movement possibly indicating suspiciousor malicious behavior on the part of the patient. Thus, in accordancewith an embodiment of the invention, if a virtual path is not properlyfollowed by the patient moving such a medication administrationapparatus, suspicious or malicious behavior may be determined

As noted above, color and/or audio sequences may also be employed.Similar positioning information may be processed relative to aninjectable medication.

In addition to determining suspicious or malicious activities takingplace within a prescribed field of view of an imaging device, and duringa time for monitoring medication adherence, various additional areas andtimes may be monitored in accordance with a preferred embodiment of theinvention. In particular, it may be possible to continue to recordbefore and after a medication administration time, thus attempting todetermine if a user removes a pill from their mouth after they thinkmonitoring has been completed, or before processing starts. If a user isto take multiple pills, for example, it is possible to continue tomonitor the user between the administration, ingestion and the like ofpills, even if the display is providing visual indications of which pillto take next, and is not displaying the face of the user. In addition,audio recording may be employed during these times to determine whetherany sounds made by the user may be indicative of suspicious or maliciousbehavior. Such audio sounds may be indicative of the user spitting out apill, actuating an inhaler or injectable device or the like. Thismonitored audio and video information may be automatically processed, inaccordance with one or more other features of the invention, todetermine whether this material provides additional evidence ofsuspicious or malicious behavior.

Finally, it may also be desirable to record a video field of view widerthan that presented to the user n a display screen. Thus, if a wideangled view of a scene is imaged, a standard vide may be displayed tothe user, and thus a user attempting to trick the system may think theyare out of the field of view of the camera because they are not shown inthe display, and may thus perform a suspicious or malicious act. If thisact takes place within the field of view of the camera, even if notshown on the display, it may be analyzed or suspicious or maliciousbehavior.

Therefore, in accordance with various embodiments of the invention,because a video image of the patient actually administering an inhalableor other medication (or other method of medication administration,including but not limited to injections, dialysis, and any othermedication administration procedure) may be captured and analyzed,analysis for suspicious or malicious behavior may be performed. Ascompared to a self reporting system or other available medicationmonitoring system, such as electronic medicine bottle caps, the variousembodiments of the present invention are the only system that allow forthe determination of one or more users trying to trick the system. Thisis because rather than simply relying on the patient to state that aparticular medication was administered, the system makes an automateddetermination of proper administration. Through the accumulation ofdata, various motions, movements or sounds may be associated withsuspicious or malicious behavior. It is only the various embodiments ofthe present invention that allow for this independent analysis of useractivity.

Such a video image may be captured or stored in any appropriate formatgiven a selected type of activity or gesture recognition that isemployed in accordance with a particular embodiment of the invention.Such may include full video, biometric data points, recording ofmovement of an article, such as a bracelet or the like, affixed to thepatient or administrator, use of mapping to provide a stick figure orother body movement tracking technique, or gesture or activityrecognition to determine movement or the like. The user may beencouraged to use a particular sequence of movement to be confirmed thatthey are properly administering the medication according to theprotocol, thus reducing the possibility of the potential appropriatemovements considered to be “correct.” Or, as noted above, capture ofcustomized video sequences may be performed so that the user is morelikely to repeat these same actions. Indeed, various instructionalvideos or other appropriate training may be provided to a user to insurethey properly administer the medication. Thus, the level of water beforeand after drinking may be automatically analyzed, determination of oneor more additional pills having been removed from a blister pack may beautomatically performed, or injectables or inhalables may beautomatically monitored to confirm consistent use, all through visualanalysis of collected video sequences.

Once the video sequences are recorded and stored, preferably in a HIPAAcompliant manner, further analysis may be performed thereon. Forexample, time on task may be measured for each portion of the medicationadministration sequence to be performed. In such a manner, it may bepossible to determine how long a particular user takes to perform aportion of the medication administration sequence. Patterns may bedetermined to identify users who perform suspicious or maliciousactivity. For example, certain actions may be performed too quickly anduniformly over time, indicating a problem. As an example, in oneparticular embodiment of the invention, the user may be asked to drink aglass of water after ingesting a pill. If the time to drink the water istoo consistently too fast, this may give evidence that the user is infact not drinking the water, but rather is faking it to trick thesystem. Other time on task measurements may also be employed. Time stampmarkers may thus be captured to confirm that the user is taking theirmedication at appropriate times and a number of times a user has taken aparticular medication, to confirm whether there are substantial delaysbetween instruction and administration, or for any other time sequencedetermination that may aid in determination of suspicious activity.Furthermore, other behavioral markers, such as, by way of example only,shaking hands indicating a nervousness, or other movements by a patientthat may give a hint as to the physical or mental status thereof, andpotential for trying to trick the system may be further employed.

The system may also preferably attempt to determine whether the use isemploying a substitute or altered medication, such as if a pill appearsdented or otherwise tampered with, has color changes, marking changes,texture changes, translucency changes, etc. A single or multipleconsecutive or non-consecutive images may be employed in order todetermine any changes in visual characteristics of the pill. Thus,confirming medication identity as well as determining deformation of,for example, a capsule (potentially indicative of taking the capsuleapart to remove the medication) may be employed. In this particularembodiment of the invention, the ability to determined such maliciousactivity may be dependent on the resolution of the imaging device, thehigher the resolution, the more complete a check for these issues may beperformed. Therefore, it is contemplated in accordance with anembodiment of the invention that as the resolution of cameras improves,and as the processing power of mobile and other imaging devicesincreases, more complete checks may be employed.

This captured adherence information, including suspicious or maliciousaction information, may be provided to a healthcare provider, clinicaltrial manager or the like through a dashboard allowing for the review ofinformation about an individual patient, entire population of patients,or demographically relevant information. Such information may beprovided to easily notify the healthcare provider, clinical trialmanager or the like of problem patients, demographic groups, medicationsor the like. One or more dashboards or other reporting mechanisms may beemployed as described in copending U.S. patent application Ser. No.13/189,518, filed Jul. 24, 2011 to Hanina et al., titled “Method andApparatus for Monitoring Medication Adherence”, the entire contentsthereof being incorporated herein by reference. Thus, any suspicious ormalicious activity information obtained in accordance with the presentinvention may be provided to one or more individuals in accordance withone or more methods or systems as described in the '518 application.

Further, once the information is captured to the dashboard, furtheranalysis may be performed that may aid in determining potentiallyintentionally non-compliant users. For example, if the user has a highpercentage of medication administration confirmations not employing theautomated visual administration system (i.e. the user tells theirhealthcare provider over the phone that they took their medication, orotherwise indicate to the automated system that they took theirmedication without employing the automated system for confirmation),this may be indicative of an attempt to trick the system. Other suchmetrics may be employed and may preferably be gleaned from collecteddata for one or more particular patient populations. As noted above, anyone or more of these situations may place the user in a different stateor category, and may therefore justify additional scrutiny from thesystem and any monitors o healthcare providers. The user may be movedinto or out of such state as noted above in accordance with actions theyperform.

Through the use of suspicious and malicious activity tracking asdescribed above, a type of administration language may be generated,allowing for extension to other patients, and also allowing forinterpretation of reason for differences from a predefined sequence by apatient. Thus, if a patient performs an action differently over time,this difference may provide insight to a compliance failure, or activeattempt to trick the system. It is further anticipated that analysis oflarge numbers of patients will allow for a more flexible system that mayrecognize more of a patient's suspicious or malicious movements, andthus may improve the ability of the system to function properly andidentify these malicious patients trying to trick the system. Filtersbased upon these accumulated results may be employed to further identifypotentially malicious users.

Therefore, in accordance with an embodiment of the invention, a user maybe requested to perform a predetermined sequence of actions designed toensure performance of medication administration, including medicationingestion of a pill or film based medication, inhalable or injectablemedication, or the like. During administration, one or more actionsassociated with the administration may be monitored to determine one ormore suspicious or malicious actions. These actions may comprisemovement of the user out of the display of the imaging device, othermovements associated with suspicious or malicious activity, such as thehand of a user covering their mouth at a critical time, indicatingpossible removal of the pill, coughing or the like at a critical time,or the like. These actions may be tracked over time to determine a habitof such action but the user to identify potential deliberatenon-compliance. Such data from multiple users may be accumulated todetermine a number of profiles that may be indicative of intentionalnon-compliance, and may be applied as filters to future action tocategorize one or more users as high risk for intentionalnon-compliance. Video imaging of the users employing the system mayfurther be stored for future analysis in order to determine time on taskdata, for example, to screen whether any particularly noticeable actionscan be determined, such as the drinking of a glass of water in too fasta time, extremely consistent time for performing each action such thatit is unlikely the user is actually taking the medication, or the like.Other metrics, such as a large number of manual skips, or otherconfirmations of medication administration not employing the inventivevideo medication administration system may also be indicative of aprofile of a user that should be considered to further scrutiny.

Further uses of the video capture sequences may also be employed,including video capture of responses to questionnaires about currentpatient states of discomfort, informed consent, and the like which maybe further used to determine whether a user is attempting to furthertrick the system. The patient may be able to send a video messageanswering one or more questions, the answers or any other motion orcharacteristic of the user when answering the questions may be furtherused to determine whether the user is attempting to trick the system.

In accordance with various embodiments of the invention, whenconsidering administration of an inhalable or injectable medication,analysis of adherence video sequences may be employed to determine alikelihood that a patient has actually administered their medication,including one or more gestures of administration of the medication.Thus, based upon video and audio cues determined related to positioningand use of the medication administration apparatus, it may be determinedthat the patient is likely to be purposefully improperly positioning theapparatus, and therefore the system may indicate that it is possiblethat the patient has administered the medication intentionallyimproperly. Low confidence in proper administration based upon failureto properly position the apparatus, failure to record audio signalsindicative of proper administration or the like may be employed todetermine whether a patient should be further watched for additionalfuture suspicious or malicious activity, via the automated trainingsystem described herein, by automated contact, or by individual personalcontact. This determination of low confidence of administration, even ifit is ultimately determined that administration likely took place, maystill be utilized to classify the user as a potentially malicious userrequiring greater scrutiny. Thus, the user may be classified as a goodpatient, a suspicious patient, malicious patient, etc., and may furtherprovide one or more notifications to a monitor, healthcare provider, orsponsor (in the case of a clinical trial). Such notifications maycomprise a dashboard notification (noted below), a text message, andautomatically generated report or the like. Such confidence levels maybe used, in accordance with a desired algorithm or the like, to providean overall picture of medication administration by a patients or groupof patients, thus allowing for further scrutiny based upon variouscharacteristics of the patient population if it appears actions arechanging, but not necessarily waiting until a critical issue isdiscovered.

It is further contemplated that the method and apparatus of theinvention allow for integration with one or more audio or videoconferencing systems, thus receiving and/or providing information therethrough. Thus, a user may employ a standard video conferencing tool orsystem, and have this information be coupled to a mobile or other devicebeing used in accordance with an embodiment of the present invention.

In addition to processing user data locally in accordance with theimaging device, this information may be transmitted to a remote locationfor processing. Thus, in accordance with one or more embodiments of theinvention, any and all processing noted herein may be performed locally,remotely, on in any combination as appropriate. The local device istherefore provided with a processor, imaging device and the likenecessary to perform all actions necessary, including transmission andreceipt of data from a remote location. Such transmission preferablytakes place over a secure, encrypted transmission system with encrypteddata. Similarly, any remote computing location preferably includesstorage, processing and transmission and reception systems sufficient toreceive, process and transmit secure data in accordance with the presentinvention. Thus communication between the local device and remotelocation is provided via a publically available network, private networkor the like. Alternatively, data may be stored on the local device andtransferred to the remote device as desired, or all processing may takeplace locally without any transmission of data to a remote location.Additionally, any type of camera may be employed, such as a cameraincluded in a mobile device, tablet or laptop computer, cellular orsmart phone and the like.

It is further contemplated in accordance with one or more embodiments ofthe invention that data provided from one or more of the above notedsystems, sensors, cameras or the like may be combined with data from anyone or more others to provided further information about suspiciousactivity. Thus, for example, synchronization of audio and video mayprovide an even better picture of malicious or suspicious behavior.

Furthermore, user of high resolution video images may also be employed.For example, if a user is determined to be at high risk for suspiciousbehavior, at one or more critical times during medicationadministration, higher resolution images may be obtained to allow for agreater level of scrutiny. By only retrieving these higher resolutionimages at the one or more critical times, overall processingrequirements are reduced while allowing for a more complete review ofuser actions. Alternative, a plurality of captured lower resolutionimages may be combined to generate a high resolution image, usingtechniques such as Super Resolution or the like. In such a manner,higher resolution images are made available to review more criticalportions of a medication administration sequence, while lower resolutionimages can be employed when suspicious activity is less likely.

In accordance with one or more embodiments of the present invention, thesystem in accordance with one or more embodiments of the invention willlearn to recognize suspicious activity and may preferably attribute ascoring system based on different metrics. As noted above, as morepatients are identified, profiles will be defined by the state machineand confidence levels attributed to patients who demonstrate theseactivities.

Various embodiments of the invention further include one or moreautomated processes for employing data mining/machine learning/AItechniques to provide a smarter system that can better classifypatients, and then more easily monitor their medication administrationover time. It is contemplated in accordance with the system that thesystem will be able to:

a) Understand the pattern of a patient with suspicious behavior

b) Classify the current patient into different categories. Such as “Gooduser”, “Suspicious User—need to be monitored”, “Suspicious User—need tonotify sponsor” and etc., thus placing the user into a particular stateand driving the engagement of the system by the user in accordance withone or more attributes of that class.

c) Suggest a way to the human or even automatically implement a way(such as sending a text message) to alert the patient about the behaviorbased on his usage pattern.

The system preferably also provides one or more abilities to togglebetween high res and low res images for capturing suspicious behaviors,as noted above. In particular this toggling may be performed by: 1)Capturing a high resolution image or a sequence of high resolutionimages at region of interest or time of interest to help check forsuspicious behavior either by human or machine; 2) During other time orregions, low resolution images may be processed to make sure that thesystem can provide real-time feedback; and 3) capture low res images,and create a high res image from a serious of low res images foranalysis on the server using techniques such Super Resolution.

Data captured by different sensors may also be fused to make a judgementwhether the patient is suspicious. The fusing of different sources ofdata may include data captured by the system and other systems orpatient information or history.

Therefore, in accordance with the invention, a method and apparatus areprovided that allow for the automated confirmation of adherence toadministration protocol for medication, and provide for a mostsophisticated method for confirming and studying methods ofadministration of such prescription medication.

It will thus be seen that the objects set forth above, among those madeapparent from the preceding description, are efficiently attained and,because certain changes may be made in carrying out the above method andin the construction(s) set forth without departing from the spirit andscope of the invention, it is intended that all matter contained in theabove description and shown in the accompanying drawings shall beinterpreted as illustrative and not in a limiting sense.

It is also to be understood that this description is intended to coverall of the generic and specific features of the invention hereindescribed and all statements of the scope of the invention which, as amatter of language, might be said to fall there between.

1.-20. (canceled)
 21. A medication administration confirmationapparatus, comprising: a display for displaying a first set of one ormore instructions encouraging proper performance of one or more steps ofa medication administration sequence; a video capture device operable tocapture one or more video sequences of a user administering medicationin response to the displayed one or more instructions; a memory operableto store the captured one or more video sequences; and a processoroperable to analyze at least one video sequence of the stored videosequences to: identify a change in one or more symptoms of the user,evaluate the identified change in one or more symptoms to identify anindication of suspicious activity, and output to the display one or morefurther instructions encouraging proper performance of the one or moresteps of the medication administration sequence in response to theidentification of one or more indications of suspicious activity. 22.The medication confirmation apparatus of claim 21, wherein the processoris operable to output a prompt, to the user, to perform a particularsequence of actions to be stored as the one or more video sequences. 23.The medication confirmation apparatus of claim 21, wherein theindication of suspicious activity comprises a face of the user leaving afield of view of the video capture device.
 24. The medicationconfirmation apparatus of claim 21, wherein the indication of suspiciousactivity comprises the hand of a user moving to their mouth.
 25. Themedication confirmation apparatus of claim 21, wherein the indication ofsuspicious activity comprises the user coughing when a medication is intheir mouth.
 26. The medication confirmation apparatus of claim 21,wherein the at least one video sequence further comprises an audiocomponent associated therewith.
 27. The medication confirmationapparatus of claim 21, wherein the video capture device furthercomprises a second display operable to display a first portion of avideo being captured.
 28. The medication confirmation apparatus of claim27, wherein the processor is operable to review a second portion of thecaptured video for suspicious behavior without outputting the secondportion to the display.
 29. The medication confirmation apparatus ofclaim 21, wherein the processor is operable to record the at least onevideo sequence after the user has administered the medication.
 30. Themedication confirmation apparatus of claim 21, wherein the change in oneor more symptoms comprises a variation in a concentration of the user.31. The medication confirmation apparatus of claim 21, wherein thechange in one or more symptoms comprises a variation in a speed of theuser to complete one or more tasks.
 32. The medication confirmationapparatus of claim 21, wherein the change in one or more symptomscomprises a variation in a motor control ability of the user.
 33. Themedication confirmation apparatus of claim 21, wherein the change in oneor more symptoms comprises a change in an eye movement pattern of theuser.
 34. The medication confirmation apparatus of claim 21, wherein theprocessor is located at a remote location.
 35. The medicationconfirmation apparatus of claim 21, wherein the processor is operable tomark the at least one video sequence as including suspicious activity.36. A medication administration confirmation apparatus, comprising: adisplay for displaying, to a user, a first set of one or moreinstructions encouraging proper performance of one or more steps of amedication administration sequence; a video capture device operable tocapture one or more video sequences of a user administering medication;a memory operable to store the captured one or more video sequences; anda processor operable to analyze at least one of the stored videosequences to: determine a change in one or more symptoms of the user,the one or more symptoms including at least one of (i) an ability of theuser to concentrate on one or more tasks, (ii) a speed with which theuser is able to complete the one or more tasks, (iii) a motor controlability of the user, (iv) an eye movement pattern of the user, or (v)mood of the user, evaluate the determined change in the one or moresymptoms to determine potentially suspicious behavior of the user, andoutput to the display one or more further instructions encouragingproper performance of the one or more steps of the medicationadministration sequence in response to the determination of theoccurrence of potentially suspicious behavior.
 37. The medicationadministration confirmation apparatus of claim 36, wherein videosequences are captured and analyzed for a plurality of users, andwherein the processor is operable to define one or more filtersapplicable to a captured video sequence to determine a change in the oneor more symptoms of the user in accordance with the analyzed videosequences from the plurality of users.
 38. A computer-implemented methodfor confirming medication administration, comprising: displaying, by amedication administration confirmation apparatus, to a user, a first setof one or more instructions encouraging proper performance of one ormore steps of a medication administration sequence; capturing one ormore video sequences of a user administering medication by a videocapture device of the medication administration confirmation apparatus;storing the one or more of the captured video sequences to anon-transitory memory of the medication administration confirmationapparatus; and analyzing, by the medication administration confirmationapparatus, at least one video sequence of the one or more stored videosequences to: identify a change in one or more symptoms of the user, andevaluate the identified change in one or more symptoms to identify anindication of suspicious activity of the user.
 39. Thecomputer-implemented method of claim 38, wherein analyzing the at leastone video sequence further comprises: identifying one or moreindications of suspicious activity from a plurality of stored videosequences from a plurality of users; generating a filter associated withone or more of the identified indications of suspicious activity; andapplying the filter to the at least one video sequence to determine afirst indication of suspicious activity.
 40. The computer-implementedmethod of claim 39, wherein the one or more symptoms including at leastone of (i) an ability of the user to concentrate on one or more tasks,(ii) a speed with which the user is able to complete the one or moretasks, (iii) a motor control ability of the user, (iv) an eye movementpattern of the user, or (v) mood of the user.